TRACX2: a RAAM-like autoencoder modeling graded chunking in infant visual-sequence learning

Even newborn infants are able to extract structure from a stream of sensory inputs and yet, how this is achieved remains largely a mystery. We present a connectionist autoencoder model, TRACX2, that learns to extract sequence structure by gradually constructing chunks, storing these chunks in a distributed manner across its synaptic weights, and recognizing these chunks when they re-occur in the input stream. Chunks are graded rather than all-or-none in nature. As chunks are learned their component parts become more and more tightly bound together. TRACX2 successfully models the data from four experiments from the infant visual statistical-learning literature, including tasks involving lowsalience embedded chunk items, part-sequences, and illusory items. The model also captures performance differences across ages through the tuning of a single learning rate parameter. These results suggest that infant statistical learning is underpinned by the same domain general learning mechanism that operates in auditory statistical learning and, potentially, in adult artificial grammar learning. Introduction We live in a world in which events evolve over time. Consequently, our senses are bombarded with information that varies sequentially over time. One of the greatest challenges for cognition is to find structure within this stream of experiences. Even newborn infants are able to do this (Teinonen, et al. 2009; Bulf, Johnson & Valenza, 2011), and yet, how this is achieved remains largely a mystery. Two possibilities have been suggested (see Theissen, et al., 2013 for a detailed discussion). The first, characterised as statistical learning, involves using frequency and transition probabilities to construct an internal representation of the regularity boundaries among elements encountered. The second possibility suggests that elements that co-occur are recalled and simply grouped together – or chunked – into single units. Over time, these chunks can themselves be grouped into super-chunks or super-units. According to this view behaviour is determined by the recognition of these chunks stored in memory and associated with particular responses. What distinguishes these accounts 1 This article is an abridged, modified version of Mareschal, D. & French, R. M. (2017) TRACX2: a connectionist autoencoder using graded chunks to model infant visual statistical learning. Phil. Trans. R. Soc. B 2017 372 20160057; DOI: 10.1098/rstb.2016.0057. is that the former argues that it is the probabilistic structure of the input sequence that is represented and stored, whereas the later argues that specific cooccurring elements are stored, rather than the overarching statistical structure. Ample evidence in support of both of these views has been reported. We will argue that this is a false dichotomy: both transitional probability learning (statistical learning) and chunking co-exist in one system that smoothly transitions between these apparent modes of behaviour. The appearance of two modes of learning is an illusion because only a single mechanism underlies sequential learning; namely, Hebbian-style learning in a partially recurrent distributed neural network. Such a system encodes exemplars (typical of chunking mechanisms) while drawing on co-occurrence statistics (typical of statistical learning models). An important corollary of this approach is that chunks are graded in nature rather than all-or-none. Moreover, interference effects between chunks will follow a similarity gradient typical of other distributed neural network memory systems. Chunks are most frequently thought of as all-ornothing items. Who thinks of "cups" and "boards" when they see the word "cupboard"? Or "foot" and "ball" when they encounter the word "football"? Indeed, chunks like these have essentially the same status as "primitive" words like "boat" or "tree", which are not made of component sub-words. But new chunks do not suddenly appear ex nihilo in language. Rather, they are generally formed gradually, their component words becoming more and more bound together with time and usage. For example, when we encounter the words "smartphone", "carwash", or "petshop", we still clearly hear the component words. We hear them less in words like "sunburn" and "heartbeat". We hear them hardly at all in "automobile." How long did it take for people to stop hearing "auto" and "mobile" when they heard or read the word "automobile"? Like "automobile", it is likely that in a few years the current generation will no longer hear "smart" and "phone" when they hear the word "smartphone". This simple observation involving the graded nature of chunks is at the heart of the chunking mechanism in TRACX2. These ideas were implicit in our initial presentation of the TRACX model (French et al., 2011). In TRACX we showed that a connectionist autoencoder, augmented with conditional recurrence, could extract chunks from a stream of sequentially presented symbols. TRACX